Checkpointing, NetCDF, and golden master tests #140

ali-ramadhan · 2019-03-19T16:42:00Z

Call them system tests, model verification tests, or end-to-end tests, that's what this PR is about.

Currently a work in progress as I hit a roadblock with checkpointing so the tests will fail. See #141. BUT, for testing we can just set model.forcings = nothing and manually reconstruct the forcings as we know them.

Tests implemented so far:

Checkpointing integration test: Run two coarse rising thermal bubble simulations and make sure that when restarting from a checkpoint, the restarted simulation matches the non-restarted simulation numerically.
NetCDF output integration test: Run a coarse thermal bubble simulation and save the output to NetCDF at the 10th time step. Then read back the output and test that it matches the model's state.
Thermal bubble golden master test: Run the coarse thermal bubble simulation for 10 time steps and check that the model output matches a golden master output.
Deep convection golden master test

@jm-c @christophernhill: do post any ideas for tests that should be implemented in this PR.

But need to be able to serialize functions first...

Helps avoid #141 when forcing functions aren't used.

This is temporary until we solve #92. At least this way you can read output and get back an array of the expected size (Nx*Ny*Nz).

codecov · 2019-03-20T00:12:12Z

Codecov Report

Merging #140 into master will increase coverage by 3.21%.
The diff coverage is 77.77%.

@@           Coverage Diff            @@
##           master   #140      +/-   ##
========================================
+ Coverage   56.78%    60%   +3.21%     
========================================
  Files          19     19              
  Lines         597    620      +23     
========================================
+ Hits          339    372      +33     
+ Misses        258    248      -10

Impacted Files	Coverage Δ
src/models.jl	`80% <ø> (ø)`	⬆️
src/output_writers.jl	`37.64% <77.77%> (+37.64%)`	⬆️
src/time_steppers.jl	`67.72% <0%> (+0.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47b4613...b880a9b. Read the comment docs.

codecov · 2019-03-20T00:12:13Z

Codecov Report

Merging #140 into master will increase coverage by 9.94%.
The diff coverage is 78.94%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
+ Coverage   56.03%   65.97%   +9.94%     
==========================================
  Files          19       19              
  Lines         605      629      +24     
==========================================
+ Hits          339      415      +76     
+ Misses        266      214      -52

Impacted Files	Coverage Δ
src/models.jl	`80% <ø> (ø)`	⬆️
src/output_writers.jl	`79.78% <78.94%> (+79.78%)`	⬆️
src/time_steppers.jl	`67.72% <0%> (+0.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7163560...371af73. Read the comment docs.

glwagner · 2019-03-20T00:51:14Z

src/output_writers.jl

    metadata, grid, stepper_tmp = model.metadata, model.grid, model.stepper_tmp
-    if metadata.arch == :cpu
+    if metadata.arch == :CPU


Why do we save FFT plans?

It makes sense that a user should plan new FFTs if they are going to run a model, even if they are restarting a checkpoint on the same computer.

@christophernhill what do you think?

We don't. We set model.poisson_solver = nothing before checkpointing then reconstruct the FFT plans afterwards.

So you're saying the user should be manually planning the FFTs themselves?

ali-ramadhan · 2019-03-20T14:35:42Z

Note to self: Seems like the golden master tests fail on Mac (Travis CI) so it seems that JLD might not be portable (although I generated the JLD checkpoint files on Windows and the Linux tests pass).

So for golden master testing I should rely on NetCDF output which is guaranteed to be portable.

EDIT: Hmmm, it might actually just be finicky (sometimes passes, sometimes doesn't). Another reason to use NetCDF. But so far if I keep running the tests on my laptop it always passes.

I think there's an issue with the pseudorandom number generator being portable. We still have the thermal bubble golden master test.

ali-ramadhan · 2019-03-21T18:38:33Z

@jm-c and I discussed this PR yesterday and seemed good to merge. Looks like the tests work now so I'm merging.

Checkpointing, NetCDF, and golden master tests Former-commit-id: ecc0e46

ali-ramadhan added 4 commits March 19, 2019 11:58

Checkpointer constructor with kwargs.

001c2d9

Fixed bug with filename function for ::Checkpointer.

f66808f

Fixed FFT plan reconstruction during checkpointing.

774a181

Basic checkpointing test.

e62c8c2

But need to be able to serialize functions first...

ali-ramadhan added the testing 🧪 Tests get priority in case of emergency evacuation label Mar 19, 2019

ali-ramadhan added this to the v0.5 milestone Mar 19, 2019

ali-ramadhan mentioned this pull request Mar 19, 2019

Checkpointer cannot serialize functions to disk with JLD. #141

Closed

ali-ramadhan added 6 commits March 19, 2019 18:22

Cooling disk temperature forcing function.

755000c

Set forcing functions to nothing when restoring.

fdaa3d7

Helps avoid #141 when forcing functions aren't used.

Thermal bubble checkpointing test works!

205aec8

Saving velocity fields at xC, yC, zC.

a3bdf43

This is temporary until we solve #92. At least this way you can read output and get back an array of the expected size (Nx*Ny*Nz).

NetCDF thermal bubble integration test.

5b3fc4c

Forgot to commit the actual test.

b880a9b

glwagner reviewed Mar 20, 2019

View reviewed changes

ali-ramadhan added 4 commits March 20, 2019 09:38

Forgot to describe the NetCDF integration test.

8786ac3

Thermal bubble golden master test.

34fca53

Thermal bubble golden master checkpoint file.

ae3e5ba

Adding Random to manifest.

5dfbb4f

ali-ramadhan added 3 commits March 20, 2019 10:40

Deep convection golden master test.

dca7290

Deep convection golden master checkpoint file.

a8b6b6c

Updating environment project manifests.

3b16880

This was referenced Mar 20, 2019

Validation and performance benchmarks. #136

Closed

Model verification tests. #81

Closed

Golden master output is now stored as NetCDF.

0d132d1

ali-ramadhan marked this pull request as ready for review March 21, 2019 14:49

ali-ramadhan added 2 commits March 21, 2019 10:51

Merge branch 'master' into system-tests

354ab07

Fixed missing async kwarg for NetCDFOutputWriter constructor

5d6cec8

Skipping deep convection golden master test.

371af73

I think there's an issue with the pseudorandom number generator being portable. We still have the thermal bubble golden master test.

ali-ramadhan self-assigned this Mar 21, 2019

ali-ramadhan changed the title ~~[WIP] System tests~~ Checkpointing, NetCDF, and golden master tests Mar 21, 2019

ali-ramadhan merged commit ecc0e46 into master Mar 21, 2019

ali-ramadhan mentioned this pull request Mar 25, 2019

z boundary conditions are being applied Nz times #150

Closed

ali-ramadhan deleted the system-tests branch March 28, 2019 13:32

arcavaliere pushed a commit to arcavaliere/Oceananigans.jl that referenced this pull request Nov 6, 2019

Merge pull request CliMA#140 from climate-machine/system-tests

67cf91b

Checkpointing, NetCDF, and golden master tests Former-commit-id: ecc0e46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Checkpointing, NetCDF, and golden master tests #140

Checkpointing, NetCDF, and golden master tests #140

ali-ramadhan commented Mar 19, 2019 •

edited

Loading

codecov bot commented Mar 20, 2019

codecov bot commented Mar 20, 2019 •

edited

Loading

glwagner Mar 20, 2019

ali-ramadhan Mar 20, 2019

ali-ramadhan commented Mar 20, 2019 •

edited

Loading

ali-ramadhan commented Mar 21, 2019

Checkpointing, NetCDF, and golden master tests #140

Checkpointing, NetCDF, and golden master tests #140

Conversation

ali-ramadhan commented Mar 19, 2019 • edited Loading

codecov bot commented Mar 20, 2019

Codecov Report

codecov bot commented Mar 20, 2019 • edited Loading

Codecov Report

glwagner Mar 20, 2019

Choose a reason for hiding this comment

ali-ramadhan Mar 20, 2019

Choose a reason for hiding this comment

ali-ramadhan commented Mar 20, 2019 • edited Loading

ali-ramadhan commented Mar 21, 2019

ali-ramadhan commented Mar 19, 2019 •

edited

Loading

codecov bot commented Mar 20, 2019 •

edited

Loading

ali-ramadhan commented Mar 20, 2019 •

edited

Loading